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ABSTRACT 

The effectiveness of an instructional improvement 
project in the Greeley-Evans Schools, Colorado, was studied. Teachers 
were provided inservice education in a class called the Essentials o£ 
Instruction. Recognizing that receiving training was not the same as 
being an implementer of the innovation, the Levels of Use instrument 
identified teachers who were or were not users of the innovation. 
California Achievement Test scores were obtained for 598 f i th grade 
students in reading, language arts, spelling, and mathr^aatics. 
Achievement gain scores were used to examine differences between 
males and females, Hispanics and Whites, and students taught by user 
versus non-user teachers. Results confirmed the achievement gains 
expected in one years* time. Males had much higher gains in reading, 
and females in spelling. Caucasians had significantly higher gains in 
mathematics and reading achievement than Hispanics did. In all 
subjects, Hispanics started with lower test scores and posted smaller 
gains than Caucasians. Results suggested that the benefits of use of 
the innovation were not fully accrued during the first year. 
Non-users showed slightly above average gains in language arts and 
mathematics, while users showed slightly above average gains in 
reading and spelling. (GDC) 
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School District #6 (Greeley-Evans , Colorado) and the University 
of Northern Colorado developed collaborative efforts to grant credit 
for knowledge, skill development, and application related to a staff 
development project labeled Success-Based Education by the school 
district. This Instructional Improvement project was began during 
Fall, 1983, and has continued to the present. 

The staff development Inservlce inputs were initially the 
responsibility of Madeline Hunter and- Sue Wells Welch. The content 
focus was in the areas of Instruction (TeachingAearning Cycle) and 
Clinical Supervision (See Appendix A and B). Subsequently, district 
personnel who were trained became district trainers. Print and 
non-print materials (video tapes and films) related to the areas of 
instruction were also used as part of the dissemination and 
implementation processes* Additionally, a class for teachers 
containing the Essentials of Instruction was developed and offered. 
Some of the district trainers also received further instruction at 
non-6reeley sites with '^unter and Wtelch. 

Initially, the evaluation plan was to assess the Impact of the 
inservlce progranmlng, as measured by attendance logs and credits 
earned by teachers, on the achievement of pupils at selected grade 
levels. Confronted with the prospect that being trained or 
knowledgable was not the same as being an implementer of the 
innovation, the validated Levels of Use (LoU) inrtrument from the 
Concerns-Based Adoption Model (CBAM) developed by the R & D Center 
at the University of Texas was used to identify "users" (See 
Appendix C). Students' academic achievement scores (Fall, 1983 and 



Fall, 1984) were obtained from the California Achievement Test 
Battery (CAT) in the areas of Reading, Language Arts, Spelling and 
Mathematics (See Appendix D). 

Di'e to the focus and level of initial training, all fifth grade 
teachers were selected for this pilot sta(fy. Teachers were 
classified as "users" and "non-users" based on interview data which 
determined the level of use of the Teaching/Learning Cycle, Of the 
24 teachers, 13 were identified as "non-users" (6 male, 7 female) 
and 11 were identified as "users" (4 male, 7 female). 

Fall '83 and Fall '84 achievement sub-test scores from the CAT 
in reading, language arts, spelling, and mathematics were obtained 
for all students in the 24 teachers' classes. Sub-test gain scores 
were analyzed to examine di.-^erences between male and female 
students, differences based on ethnicity, and differences between 
students taught by "user" teachers and "non-user" teachers. 

Pupils with the following characteristics were dropped from the 
study: special education designation, recipients of home-bound 
inscruction, non-English speakers, pupils with severe non-attendance 
records and pupils of teachers who had severe non-attendance records. 

The following primary hypotheses provided the bases for 
preliminary data analysis, 

1. There will be a greater gain in pupil achievement in all 
sub-test areas for pupils taught by "user" teachers than 
for pupils taught by "non-user" teachers. 
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Z. There will be no difference in pupil achievement gains in 
all sub-test areas for pupils taught by "user" and 
non-user" teachers when pupils are compared on the 
variable of gender, 

3. There will be no difference in pupil achievement gains in 
all sub-test areas for pupils taught by "user" and 
non-user" teachers when pupils are compared on the 
variable of ethnicity. 

Fifth Grade Subjects 

There were 598 pupils in this study. The balance between male 
and female subjects is presented in Table 1. 

Ethnicity is presented in Table 2. The most recent census data 
indicates that the Hispanic population in Weld County is 18 
percent. A very small number of other ethnic minorities were found 
in the total population studies. These have been included in the 
Caucasian population reported in Table 2. 

Test Data 

The entire population of subjects studied maintained the 
achievement gains expected in one year's time in the four sub-test 
areas. This is shown in Table 3. 

A review of the gain scores by sex revealed some interesting 
differences. A difference of approximately .20 is significant at 
p .05 for groups of 200 to 400 subjects. (See Table 4) 



5 



While there is no available explanation of the unusual average 
gain made by males as compared to females in the area of Reading, 
and females as compared to males in the area of Spelling, the 
difference nevertheless is significant. Further investigation of 
the occurrance will be conducted as additional data become available. 

In all cases tho subjects showed average gains in tlie four 
sibject areas, and all are nearly one years growth, except in the 
area of Mathematics as shown in Table 5. 

In all areas reviewed, the Hispanic population started with 
lower test scores (1983) and posted smaller gains in each area than 
the Caucasian population. The largest gain area for Caucasians was 
Mathematics. This difference is not explicidble based on the data 
available. A more detailed investigation of this is warranted. The 
next largest gain area for Caucasions was in Reading. This may be 
due to linguistic difference in the predominant language spoken at 
home. The similiarily in gains in the area of Spelling may be due 
to the nearly uniform curriculum and instructional methodology now 
being used by the school district. This irrbedded instructional 
approach may not have been altered by the alternative instructional 
approach. 

Mean differences of approximately .20 are significant at p = 
.05 for groups of 200 to 400. Given that the N = 417 for the 
Caucasian population and N = 181 for the Hispanic population (See 
Table 2} the differences in gains between the two populations in the 
area of Mathematics and Reading are significant. 
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Schools with the highest Hispanic populations were grouped 
(Group I and II) and compared to schools with three lower densities 
of Hispanic populations. Group Three and Four represents groupings 
of schools with the lowest Hispanic populations (Grrup III and IV). 
(See Table 6) 

Mean differences of aoproxlmatefy .50 are significant for 
groups of approxinately 50 subjects with p = .05. 

Test scores for 1983 appear comparable within groups and ascend 
from the lowest scores in Group One to the highest scores in Group 
Four. Differences in gains from one testing period to the next 
follow the same pattern except for School D. Sdiool D is judged by 
district officials be the school with the greatest immersion In 
staff development for the Essentials of Instruction. The fifth 
grade teachers at School D were determined to be at levels IVB and V 
meaning they were in the refinement and integration stages of 
adopting the innovation (See ^pendix C). When School D is compared 
to the other schools in Group Tno the differences in gain scores are 
significant for School D compared to School E in the area of Reading 
and for School D compared to School C in Mathematics. It is also 
noteable that the test scores in the areas of Mathematics and 
Reading for School D /Group Two compare very favorably to test 
scores, in the same areas, attained by schools in Group Four. 
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Test scores for users and non-users of the Essentials of 
Instruction are shown in Table 7. (see Appendix C for the 
behavioral indices for users and non-users.) The average 
achievement gain per year (K-4 = 5 yrs of schooling and K-5 = 6 
years of schooling) is compared to gain made during the testing 
period. One can find the difference between previous average gains 
and gain attained between 1983 and 1984. Non-users showed slightly 
above average gains in Language Arts and Mathematics while users 
showed slightly above average gains in Reading and Spelling. In no 
areas were the 1983-84 gains significantly different from the 
average gain students made over previous years of sdiool. It is 
suggested that the impact of the innovation was not fully accrued 
during the initial year of implementation. 

Table 8 displays the gain scores for the Caucasians and 
Hispanic populations. It is noteable that the differences in the 
area of Spe'"ling are minor. Again, this may be due to the 
standardization of curriculum and, instructional approach in the 
district in this area. The differences found in the area of 
Language Arts are also negligible. Fhe differences in the areas of 
Mathematics and Reading are approaching significance. Any furtiier 
attempt to stratify the population beyond sex and etfinic 
differences, i.e., user and non-user classrooms or group students by 
level of school (Group One, Two, Three, or Four) produced such small 
numbers of subjects in classes that such an analysis was of 
questionable value in the case of the Hispanic female. 
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While most research indicates that the fenale motivational 
level In the 5th grade is higher than that of the male, this studly 
may be showing subject area differences are more dependent on 
ethnicity than on gender in the area of Reading and Mathematics. In 
the Caucasian population, for three of the four areas studied the 
opposite is trua. Of course, there may be other cultural or SES 
differences accounting for these variances, such as the dominance of 
ffeld dependence Oi* field independent learning, family expectational 
differences of the learner based on gender, physiological 
maturatfonal differences, and the like. 

Summary 

In recognition of the fact that the use of Essentials of 
Instruction began in Fall, 1983, it would seem plausible that tte 
impact of such instruction had not yet fully accrued. Also, initial 
training was focused in schools with historically low achievement 
and high Hispanic ethnicity. This may well account for no 
differences found between users and non-users. 

As far as district results are concerned, the overall 
achievement is one full grade level. Overall, females tend to 
achieve some what higher than males. This may be due to differences 
in motivation based on gender. Only in the area^of Language Arts 
did most students achieved at or near grade level. It may be 
possible that the use of the Essentials of Instruction suppresses 
motivation in females and/or enhances motivation in males. 
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School D (Table 7) achievement gains in Reading and Mathematics 
are significantly higher than those of other schools in Group Two 
and certain schools in other (two higher) groupings. Since School D 
was the pilot school in which the Essentials of I/istruction program 
should have had the greatest impact, there is sufficient 
encouragement to proceed with further stuc|y of the variables in this 
initial design. 

Recommendations 

1 . The use of the LoU as a valid measure of the util ization of the 
Essentials of Instruction by teachers should be supplemented by 
principals' perceptions based on direct observation, and 
self-report by the teachers. Also, user and non-user levels 
may be redefined by placing the cut-point between Level III and 
IVA rather than III and II. 

2. The similarities of scores in the area of Spelling should be 
investigated as to possible reasons for highly similar scores 
across all schools at the fifth grade level. 

3. An indepth stuc|y of School D as to teachers aptitude and 
attitude as well as administrative leadership is merited to 
determine the possible variables contributing to tJie 
significant gain scores identified. 

5. The case of aptitude (IQ) and learning style of pupils as 
additional variables is advisable. This is currently being 
planned. 
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6. Investigation should be conducted to identify reasons for tiie 
possible differences in scores between male, Hispanic and 
Caucasian populations in the area of Katheniatics. 

7. While test scores, beginning and gain scores, for the Hispanic 
population is noticeably lower than the same scores for the 
Caucasian population, longitudinal studies should be conducted 
to determine if relative gain scores are increasing that could 
be attributable to the new instructional methodology. 
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TABLES 



Table 1 

Fifth Grade Population by Sex 

Sex Frequency PfL't?^ 

Female 289 48.3 

Male 309 51 .7 



Table 2 

Fifth Grade Population bv Ethnicity 

Ethnic F requency Percent 

Caucasian 417 69.7 

Hispanic 181 30.3 
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Table 3 

Fifth Grade Test Score Gains 
Fall. 1983/Fan, 1984 

Variable Mean Gain 



READ 83 5.72 

READ 84 6.89 +1 .17 

LA 83 5.49 

LA 84 6.62 +1.13 

SPELL 83 5.34 

SFiLL 84 6.36 +1 .02 

MATH 83 5.27 

MATH 84 6.29 +1 .02 
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Table 4 

Fifth Grade Test Score Differences by Sex 
Fall. 1983/Fall, 1984 



Females (289) Males (309) 

Variable Mean Gain Mean Gain Difference in Gains 

READ 83 5.89 5.5/ 

+1.0'i +1.29 
READ 84 6.90 6.88 (+.26 male) 



LA 83 5.77 5.23 

+1 .21 +1 .06 

LA 84 6.98 6 29 (+.15 female) 



SPELL 83 5.63 5.06 

+1 .13 + .92 

SPELL 84 6.76 * ' 5.98 * (+.21 female) 



MATH 83 5.32 5.22 

+1 .03 +1 .00 

MATH 84 6.35 6.22 (+.03 female) 
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Table 5 

Fifth Grade Test Score Difference 
Reporting Ethnicity 



Caucasian (41 7) Hispanic (181 ) 

Variable Mean Qain Mean Gain Difference in Gains 

RZPD 83 6.24 4.53 

+1.26 +.95 
READ 84 7.50 5.48 (+ .31 Cau) 



LA 83 5.94 4.47 

+1.16 . +1.04 

LA M 7.11 5.51 (+ .12 Cau) 



SPELL 83 5.66 4.60 

+1 .03 + .98 

SPELL 84 6.69 5.58 (+ .05 Cau) 



MATH 83 5.46 4.83 

+1 .13 + .77 

MATH 84 6.5 9 5.60 (+ .46 Cau) 



Table 6 

Fifth Grade Test Score Difference Among Schools 
Group by Density of Ethnic Population 

Group One 

School A School B 



Variable 


N 


Mean 


Diff 


Mean 


N Diff 




Max. Diff. 




READ 83 
READ 84 
MATH 83 
MATH 84 


33 
32 
33 
33 


4.03 
4.93 
4.65 
5.41 


+.90 
+.76 


4.03 
5.07 
4.73 
5.48 


65 

67 +1.04 
65 

67 +.75 




+.14 (School 
+.01 (School 


B) 
A) 




School 


C 




Group Two 
School D School 


E 






Variable 


N 


Mean 


Diff 


Mean 


N Diff Mean N 


Diff 


Max. Diff. 




READ 83 
READ 84 
MATH 83 
MATH 84 


30 
30 
30 
30 


5.20 
6.U 
5.12 
5.64 


+.93 
+.52 


5.19 
6.63 
4.68 
5.37 


48 5.00 46 
48 +1 44* 5.80 46 
48 4.96 46 
48 +1 .19** 5.82 46 


+.80 
+.90 


+.64 (School 
+.67 (School 


D) 
D 




School 


F 




Group Three 
School G 








Variable 


N 


Mean 


Diff 


Mean 


N D<ff 




Max. Diff. 




READ 83 
READ 84 
MATH 83 
MATH 84 


60 
61 
60 
61 


5.86 
7.12 
5.48 
6.36 


+1.26 
+ .88 


5.93 
7.07 
5.19 
5.97 


50 

49 +1 .14 
50 

49 + .78 




+.12 (School 
+.10 (School 


F) 
F) 


School H 




School I 




Group Four 
Scftool J 


School K 






Variable 


N Mean 


Diff 


Mean N 


Diff 


Mean N Diff Mean 


N Diff. 


Max. Diff. 





REM) 83 67 6.82 6.31 55 6.72 70 6.34 71 

READ 84 67 8.09 +1 .27* 7.81 54 +1 .50* 8.01 69 +1 .29* 7.88 71 1.54* +.27 (School K) 

MATH 83 67 5.82 5.30 55 5.92 70 5.40 71 

' MATH 84 67 7.07 +1 .25** 6.49 55 +1 .19** 7.17 69 +1 .25** 6.70 71 1 .30** +.11 (School K) 



Table 7 



Comparison of Gain Scores and Average Gain 
Scores Between Uses and Non-Users 



Va riable 

wsrss- 



N 

W 



Mean 



Mon-Users 

Diff K-4 K-5 Wff. wrt Ave K-5 Gain 



READ 84 



346 7.40 +1.21 



1.23 



-.02 



LA 83 
LA 84 



345 5.88 

347 7.13 +1.25 



1.18 



1 .19 



+.06 



SPELL 83 346 5.61 

SPELL 84 346 6.58 +.97 



1.12 



1.10 



-.13 



MATH 83 347 5.50 

MATH 84 347 6.64 +1 .14 



1.10 



1 .11 



Users 



Variable N Mean Kff 

RiAirss" i48 'nn 

READ 84 247 6.18 +1 .11 



K-4 K-5 D iff. wrt Ave K-5 Gain 
TTITl 



1.03 



+.08 



LA 83 
LA 84 



250 4.96 

247 5.92 + .96 



.99 



.99 



-.03 



SPELL 83 2 50 4.96 

SPELL 84 247 6 05 +1 .09 



.99 



1.01 



+.08 



MATH 83 248 4.95 

MATH 84 249 5.80 + .85 



.99 



.97 



-.12 



erJc 



19 



Table 8 

Test Score Gains 
by Sex and Ethnicity 

Math Gain 





Caucasian 


N 


Hispanic 


N 


Diff. 


Female 


1 .16 


in 

Z lU 


.70 


77 


J. AC 

+.46 


Male 


1.11 


206 


.78 


103 


+. 33 








Reading Gain 








Caucasian 


N 


Hispanic 


N 


Diff. 




1 .15 


ZIO 


.75 


77 


+•40 


Male 


1.42 


206 


1.04 


103 


• • * 








Language Arts Gain 






Ca'jcasian 


N 


Hispanic 


N 


Diff. 


Female 


1.25 


210 


1.12 


77 


+.13 


Male 


1.01 


206 


.95 


103 


+.16 








Spell ing Gain 








Caucasian 


N 


Hispanic 


N 


Diff. 


Female 


1.12 


210 


1 .10 


77 


+.02 


Male 


.94 


206 


.85 


103 


+.09 
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STEP ONE 



STEP TWO — 



STEP THREE - 



STEP FOUR - 



STEP FIVE - 



STEP SIX — 



STEP SEVEN - 



STEP EIGHT - 



STEP NINE - 



Append-ix A 
Teacning /Learning Cycle 
Lesson Design 

MENTAL SET 

Planned activity for a lesson which introduces 
the objective, focuses the learner's attention 
on thy lesson topic, or sets the learner to 
anticipate the lesson 

RATIONALE ("Need to know") 

Pat't of a 1 esson which gives learners a reason 

for accomplishing the objective 

OBJECTIVE (Content & Behavior; To be Made 
PvbUc ) 

A Statement defining what the learner is to 
learn (content) and what the learner must do to 
demonstrate that the objective has been 
mastered (behavior) 

TEACH 

Tliat part of a lesson which gives the learner 
the information needed to master the objective. 

MODEL 

The part of a lesson where a demonstration of 
expected learner behavior is provided. 
(Together, Step Four and Five uses exploration, 
exposition and/ or demonstration) 

GUIDED PRACTICE (Not graded) 

Thft part of the lesson where the teachar is 

closely monitoring learner performance to 

determine if the skill is being practiced 

correctly. 

CLOSURE (Chedcing for understanding) 

Feedback from learners at the end of a lesson 
or practice to determine if learners understand 
the learning. 

INDEPENDENT PRACTICE (Specific i immediate 
feedback } 

Practice without direction or assistance after 
adequate guided practice. 

PROGRESS CHECK 

The part of the lesson where a test or check of 
some kind is used to determine the learner's 
mastery of the objective. 
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^pend-ix B 
Steps In Clinical Supervision 

STEP ONE: Readiness 

Prepare teacher for process to be used; 
preparation of staff; guidelines must be known 
and understood by faculty; no surprises. 

STEP TWO: D I AGWOS ING THE TEACH ING 

Take notes/collect objective data; record 
exactly vihat Is going on In the classroom; 
diagnosing Instruction openly; look at 
objectives and mind set; then make quality 
judgnents based on whether It Is Important If 
component was deleted or added. 

STEP THREE: SELECT CONFERENCE PRIORITIES 

Fornative evaluation; change behavior; 
selection of component which will make a 
difference In learning; eliminate trivia; major 
element which will help across all lessons; 
limit nuni>er of change items. 

STEP FOUR: P LAN CONFERENCE 

Introduction 

I. Questions for teachers; how do they 
perceive lesson taught? 

II. Specific examples of strengths; rationale 
for why teacher should continue to 
utilize these strengths; obtain feedback 
from teacher relative to hearing what has 
been said. 

III. Specific examples of components which 
nesd work; Do NOT try U get "blood from 
a turnip"; let the teacher determine 
options which will help lessons; 
supervisor can add suggestions. 

STEP FIVE: CONDUCTING THE CONFERENCE 

T 

E 
A 
C 
H 

Must have Integrated plan. 
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Appendix C 
Levels of Use of the Innovation: 
Typical Behaviors 



Level of Use 



Behavioral Indices of Level 



rvi 



Renewal 



tn 
3 



IVB 



IVA 



Mil 



r" 

u 

<D T 
I 

g 



Integration 

Refinement 
Routine 

Mechanical Use 

Preparation 
Orientation 

Nonuse 



The user is seeking more 
effective alternatives to the 
established use of the 
innovation. 

The user is making deliberate 
efforts to coordinate with 
others in using the innovation. 

The user is making changes to 
increase outcomes. 

The user is making few or no 
changes and has an established 
pattern of use. 

The use** is making changes to 
better organize use of the 
Inn ova ti on 

The individual is preparing to 
use the innovation. 

The individual is seeking 
information about the 
innovation. 

No action is being taken with 
respect to the innovation. 



CB/IM Project 

Research and Development Center for Teacher Education 
The Uhiversfty of Texas at Austin 
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Appendix U 
Student Information Sheet 



Name , , Birth Date / / Teacher/Grade Level /Schcols 

(last) (int) (int) mo W yr 
Sex Feral e Male Aptitude ( I. Q.) Measure 1st / / 

Ethn1c1ty/SES Lm. Style/ Personality 2nd / / 

3rd / / 

TEST DATA 4th / / 

Area FaTl/83 Fall/84 Spg/85 Fan/85 Spg/86 5th / / 

Reading — 6th / / 

Ung. Arts - 

Spelling— 

Math 

Special Use Populations 
(Check yes or no) 

Yes to 

Special education pupil 

Home-bound Instruction 

Non-English native speaker 

Severe non-attendance (pupil ) (If more than 35 days of absence) 

Severe non-attendance (teacher) (If more than 25 days of absence but 

not Including Inservlce/ staff development days) 

Student receiving Chapter I assistance 



ERIC 



25 



